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7 Preface 
Many of. my thoughts about evaluation which are expressed in this 
2ck oe : 
paper have betn influenced by myeinteractions “with ‘my colleagues in 
\ : 


_-the Southeast Alternatives Projects To them, especially to those on 


the ‘Internal Evaluation Team, I wish to express my gratitude. Of 
course, T-alone am‘ respdénsible for any errors of omission or com- 
‘mission.’ ~ 
A’special thanks to Susie Demet for her editorial assistance and 
sat = to Rose ‘Salo and Lois Caswell for their clerical assistarice, 
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. Introduction 


I believe that educational evaluation as practiced in the major- 
ity of instances has failed to live up to peoples! expectations.: It 


has failed to give practitioners the information they needed to im- 


‘prove education. Educators, especially those involved in developing 


alternative programs, have come to vigw evaluation as a process having 


little value to them; indeed, in many: cases, they view it as antithet- 


ical to their purposes. .  # 

Can evaluation be improved to mnke it a worthwhile endeavor? 
Participants in the Southeast ‘Alternatives project: do believe that 
svaluation is worthwhile--that it does provide then with nsetia in- 
formatfon. Evaluators in the project believe that much of the com- 
monly aniounibercd negativism toward evaluation is the result of mis- 
advice and misunderstanding arising out of the kind-of evaluator-prac- 


titioner relationships which were noted by the Phi Delta Kappa National 


Study Committee on Evaluation (Stufflebeam, Foley, Gephart, Guba, 
Hammond, Merriman % Provus, 1971) when they said: 


Professional evaluators can no longer afford to give the practi- . 
tioner the cavalier, arrogant and condescending treatment that 
has so often characterized @heir relationships in the past (p. h). 


SEA evaluators believe that by working with decision makers rather 


than treating ‘them 3 the way the committee has noted we can greatly re- 
duce or eliminate this nagativdaa: The tipdack clner dens forth many of 
the SEA views about formative evaluation niet how we believe it should be 
practiced in said ta provide decision makers with useful information. 


While the paper is focused specifically on ‘the formative evaluation of 


' 
+ a 


alternative programs most of the suggestions presented seem equally ap- 


plicable to summative efforts. 


’ 


The Role of Internal Evaluation 


rst of all, the question of what purposes evaluation of alter- 


native prograns must and can serve should be considered. Mike Hickey 


(1972) has Stated four specific purposes of evaluation in alternative 
} 
programs: 


First, and perhaps of highest priority, is the purpose of! in- 
ternal self-improvement -for the program, which in ,turn relates . 
to the ongoing planning process (informal though it may ‘be ). 


Second, as a basis for establishing the credibility of the al- 
ternative program, evaluation must meet the demands of a vari- 
ety of "publics", , .., ' 

Third, a primary rationale forthe existence of alternatives 
within public education is that they become the means or ithe 
process by which public education evolves. Realistically, 
some strategies for educational alternatives will not work. 
Evaluation provides a base for jidentifying those that ae 
and tHose that don't. ... ; 


Finally, the evaluation of studeht progress is difficult 
without an adequate understanding of.where the program it- 
self stands (p. 2). 


@he National Institute of Educatiqn, apparently recognizing the 


value of the formative evaluation which\Hickey refers to in his first 
goal, has provided within the Experimenthl Schools Program a local 
project evaluation comfonent. According \to the NIE this local com- 


ponent 
provides for a legitimate, internalizkd role of evaluation, 
and can be tailored specifically to aw individual project and 
its goals. It should provide for quidk feedback and enhance 
resident and staff participation. It \is designed to aid in ~ 
meeting objectives and improving perfofmance. (Memorandum 
‘from R. Binswanger to J. Kent, April 13, 197h). 
s t ° 


In order-for the local’ SEAyevaluation 
that charge from the NIE, nearly all evaluat 
*’ ponent conducts are of a formative nature. formative evaluation of 


t this type may be characterized as follows: 
- 


its purpose is to improve ‘the program; , 
its primary audience is always the prudtan gartlolparitaes 
students, parents and school personnel} , a 
different avelaniten situations dictate the conditions of 
reporting--to whom, report medium, type of recommendations, 
etc; and * , 
the evaluators must be responsive ae needs of the pro- 
gram personnel. = 
Hickey's other three purposes seem to call for the eve tution 


. 


information used in a more summative sefise. Certainly, these are in- | 


portant but it is wl tons purpose of the present paper to discuss 


ways of successfully implementing a formative effort that can meet the 


information needs of decision makers. 
R 


Providing Effective Service 


It seems that one,of the major reasons for the failure of ‘evalua- 


e 


tion to provide decisiori makers with the informatign they heed to make 


RNS 
edugation better has been the lack of adequate communicution between 


evaluators and decision’ makers. Not simply failure of evaluators to 
‘ : 
effectively communicate results but their failure to involve deci- 


sion makers at every step of the process from deciding what questions. 


should be asked to interpreting the results. 


A recent study (Lange & Speiss, 1974) has shown that of four fac- 


tors related :+to communication between vans eee decision maker-- 
frequency of interaction, interpretability of reports, advocacy for 

* 4 
his conclusions, and credibility ‘of the evaluator--credibflity is the 


most important factor in determining whether or not practitioners will 


ry 
t 


use the results of an evaluation study. The SHA evaluation com- 
ponent has long believed in an approach that stresses credibility. 
Many of the past short-comings of evaluation have led prat- 


'"» 
titioners to view evaluators as uncredible. If evaluators are to 


overcome this image, they must be aware of these past short-comings 
and of possible ways to eliminate them. I want to note what I be- 


Q. 
lieve are many of the reasons for this lack of credibility and what 


SEA evaluators believe might be done to remedy the situation. * 


pt 


Past Shortcomings and Hopeful Remedies\ \ 
a 
Perhaps part of the fal ae aly a which practitioners have 


viewed evaluation has been due to: the heavy emphasis in program eval- 
uation on summative efforts, Formative evaluation efforts designed to 
ae 
provide information to ussigt in making ongoing changes have, in the 
past, been centered cduantie in the areas of student evaluation and 
curriculum (product) development. Only récently have there seemingly 
been significant attempts to extend these formative efforts to pro- 
gram evaluations in order to provide’ practitioners with information 
which can be, utilized to improve the program as it evolves. Past sun-- 
native efforts with feedback eoning, the end of a project and with 
emphasis on providing only an ovérall assessment of effectiveness have 
failed to provide this needed feedback. These summative evaluations 
have been classified as "good" only jwhen they developed no negative fine 


> 


formation. ms contrast, in formative evaluation efforts negative data 
is usually more productive than positive data since it points to areas 
of the program which Have weaknesses at a time while it is still pos- 

sible to make program ncditteatioss in an attempt to. eliminate these 


weaknesses. ¢ 


It seems that both decision mitors and evaluator’ nust share 


the blame for the lack 9 tive evaluat ion efforts in thé past 
/ , 4 
but now both groups must work together tr 


Ns sod : 
internal formative evaluation offort is included as. an,integr 
. ® s © 


' of every program development effort. Evaluation can, ese 
ae Z = * . 
practiced in a formative sense, provide the answers to many Of the 


decision makers questions; to avoid it hecause it has failed in the 


7 


past is a "cop-ont." ‘ 

Another major oe much of ‘the’ negativism toward evalua~ 
tion may very well be due to a lack of understanding of what eval+ 
uation is all about. Mike Hickey (1972) has noted this as follows; 


much of the hostility toward evaluation has arisen from 
failure to understand what evaluation is all abou$. The 
‘blame for this failure rests“equatly with both sides: the 
evaluator for not clarifying what he is doing and why, and 
for imposing an evaluation design on the program; the pro- | 
gram director fomnot requiring full explanation of the pro- 
cess and its underlying rationale and for not taking the ini- 
tive for developing, at Jdast partially, the evaluation de- 
sign (p. 1). ' : 


Probably Targely as -a result of this failure of project per- 
sonnel to demand an explanation of the process, much of what has 
been passing as piuaaiane has not, been aration but has, rather, 
been research. Research methodology and evaluation methodclogy are 
not the same and attempts to equate 
“them forces certain constraints inimical to the purposes of 
evaluation and makes it impossible to meet certain of the needs 
served by yood evaluation (Stufflebeam et al., 1971, pe 22). 
The purpose of ‘research is tig produce néw knowledge that is’ gen- 
“eralizable and classic research design procedures call for experimental - 
control of extraneous variables to accomplish this goal. In evaluation 


‘the goal should -be to delineate, obtain and provide information for de- 


cision making. This information is not necessarily new knowledge and 


ae 


45 highly specific to a given situation. The decision maker neds : 
data on how welt niaghor goals are being achieved in one real life 
situation, not on what would ‘happen in a "laboratory" situatied with 
Ahertesenae from aeipanague variables elininated. 


As there are differences in the: design* of research and evaluation 


studies there are also different requirements as far as the kinds of 


measurement instruments and observation techniques that are needed. 


, = * 
ry 


As Rosen (1973) has noted: . 2 


Experimenta] research always requires valid and reliable 
measurement instruments. Validity and reliability are also 
important criteria for evaluation instruments, but as the 
purpose of evaluation is to improve an enterprise, the eval- 
ud4tor cannot. limit herself 'to measuring only variables for 
which there are tested instruments and techniques (p. hi). 


Certainly, SEA evaluators believe that validity is important--in 


fact, one of the majo. prublems in using only instruments and tech- _ 


. 


niques that’ have Leen shown to be reliable is that they do not produce 


valid snswers to many of the questions important to decision makers. 


However, evaluators should not be expected to conduct statistical valid- . 

ity studies on every instrumem or technique they develop. ‘Face validity 
® 

assessments must ‘suffice. 


Also, as evaluators develop finctruments or techniques they must ‘ 


follow sound practice to achieve devices that, will produce reliable in- 


formation but, again, ‘they mist nqt be expected to conduct reliability 
studies for every device. 


Duenbhas4aing the importance of reliability and statistical valid- 


’ 


ity seems especially appropriate to formative program evaluation efforts. 


As opposed to many summative evaluations, irrevocable decisions are not 


being. made in which the life of the program is at stake. The formative 


sd 


information should be used to as sist the decision maker in determining 


needed program changes. If these changes are made and do not "pan-out", . 
: possibly as a result of unreliability or invalidity of information, // 


: continual formative monitoring will allow corrective action to be taken 


quickly. . % ‘ re ae 


- Attempts to deal with’ evaluation as if it weré research has also 
lead to a tendency to cast Aiwivablion studies jn the classic Aeiebs 
mental/control group mold so ‘common in research. This has been done 
in'spite of the fact that as early as 1963 ‘Cronbach: stated that: 

"The aim to compare one course with another should not dominate plans Ne 
for evaluation tay 13)." If evaluation information is to be aati 46 +s 
the decision maker s/he shotrld be concerned with how the program is wg 
meeting 1e6 goals, not how it compares with some competing program. 

The important thing is for the evaluator to look at the ‘sacitnatediae 


ing process and to provide information to enable the decision maker to 


strengthen weak aspects of the program. 


Many of the problems which arise from the failure to recognize the * 
‘ “y 
distinctly different purposes of research and evaluation studies flay 
’ ‘ 
well ‘stem from the training of most early evaluators. When the need: 


for a large number of evaluators first arose the majority of those 

drawn into the field had strong roots in the areas of psychological 

‘and educational research. They were steeped in the methods of classical 
research and tried to "bend" evaluations to fi these methods rather 
than to adopt new more appranndate methods . In the past decade, partic- 
ularly in the ‘past five years, enough has been written of the differ- 
ences} but, evaluators trained in appropriate approaches are still not 


‘ 


readily available. Practitioners must demand evaluators who are evalua- 


fers not researchers and they Will get them, : ; ¢ 


é 
«+ Another reason why evaluators. often fail to answer many of the 


questions that are important to’ decision makers hag Pea stated as 


certs ae ; 4 


With their attention riveted on sitet evaments’ evaluators over- 
looked the process because’ the really important data, product 

- data, would be available only at the end, when student behaviors 
could be checked against the objectives (Stufflebeam et al., 
1971, pe 13) 


' This failure of evaluators to consider the process hts -been in- 
f . : 


exorably- linked to their insistence that the evaluation process be cen- 


tered around previously defined behavioral objectives. for the ‘program. 


w 
To be sure, goals and objectives -are important but evaluators must 


come to realize that nob everything can properly be assessed in terms 
\. 
of student product data. Particularly, in formative ‘evaluation ef- 
’ i 
forts evaluators must consider the process gs more important than the 


prodict. 

Another plousderaianaina that has tead to many problems has been 
the failure of evaluators to discuss with practitioners the different 
ways in which evaluation may be defined, to make thefM aware of the con- 
sequences of the sa definitions and for evaluators and practition- 
ers to jointly) agree on * whieh definition should be adopted for a parti- 
gular task. Two common definitions isinirisbeek et al., 1971 & Gephart, 
1973) and their implications are as follows: 

Evalation is identical to measurement. ous approach igvolves ad~ va 


nlinistering a written nogm-referenced test ‘and interpreting the scores 


* ¥ 


as the evaluation. This’ “adnan a seemingly deisctive way to evaluate 
because the scores are acndenad to a norm. “£ “But, this obscures the fact 
that, pinnae somewhere Es progran decksica making value judgments 

are necessary. Sinee tthiis method amas bnlly on written norm-referenced 


tests, it has resulted in eviiuatlons that, have es narrow in facus. 


-It has hot dleatt effectively with evaluation of } 
"and affective learning. with its emphasis on outcome measures it has 
alsa failed to provide evaluation of. the téaching/Leafning process. 
Evaluation of: the PERSESE is especially appre BE ‘evelust oe? étfonte 
are to provide the feedback necessary to assist in the ongoing program 
development. ieluatova can and should use a . variety of methods for: 
gathering information--tests (mostly objectives based), surveys, in- 
terviews, observations, etc. Many of these methods are now Seneben 
off because they lack reliability; but, as noted earlier, this should 
“be considered grounds for exclusion. It goes without saying that not 
all of these methods are sohenidate in.a given situation but they are 


s 


all tools which should ‘be considered for appropriateness. ins 


Evaluation equals judgment. In. oe apprend an expert is Bepee 
to judge the progran. The drawbacks of “the approach are many « The 
evaluator is making impressionistic judgments with the data and the 
criteria used to assess them hidden dnd it is difficult fo dotacniie 


the validity of the judgments. The ert often does not systemati- 


cally consider the decision maker's goals so the “is frequently ir- 
eleven. Most importantly, since this type of evaluation ‘presen 
judgments rather than data to the practitioner it takes the decision 
making process (at least much of it) out of the hands’ of the practi~’ e 
tioner and places it in the hands of the evaluator. 

The evaluation is identical to measurement and the evaluation is. 


iuigsent spprdaches used alone, seem to have unsurmountable validity : ( 


a 


pabieies pee for alternative schools in which major emphasis 


is placed upon more than low order cognitive learning and in decentral- 
; ( 
izing the decision making process. In the SEA project the main purpose 


of internal evaluation is to assist decision makers. Thus, project in- 


° au 12 - 


, . 


ternal evalugtion efforts fall under a final definition given by Gephart , 


‘ee (2973). 3 


bk ae Evaluation serves depision making. Under..this ye nnn evalyd- 
"tion is considered as/an integral part. “of the decision naling <2 


'. The ‘spécific decision to be nade gives direction to the evaluation éf- 
g fort and evaluation provides information to assist in making th deci- 


. 


sion but. it does not make the decision. | ae : : “8 5S 


. 


romisinj mode t Operating under Copharts definition SEA. e aie 
say rs believe decision nakciig/evaliuation should follow a process |simi- 


lar to the following (Rawitsch, 197): 


h 


r ° "Given: We must make a decision. To. make this decision, we need ~ 


* the answérs to certain qu uestions. To get the ‘answers to 

ae . these questions, we need certain information. To get this 
inforhation, we need "data" gathering strategies... (p. h). 

thud thetspecitic decision, to be made provides aa a: for the 


Ava Saties phones which cindists of the three steps listed. ‘It should - ‘ 


be hele that every decision maker: evaluates whether formally or in- o 


aps “That ‘is, any Lime s/he must make a Santpion s/he will” form, e 


qu stions ‘and will seek wers to these’ questions og sae some ee 


strategy to gather information--even if it is a very loosd and infor- 
\ 5 cee 
‘ mal airELEEy: ; ee - ns a 
When someone called Nevalluator" is called in to help with, Mie a0 


- 


& ; process Foahnia motion s to assist \the decision maker(s), the aes 
‘uator becomes in essence an extension of the decision makers mental’ pro= 
cess: The evaluator helps the decision maker (s )° by” 
. “sdelinesting--helping the decision maker determine what aatanas 
g : ‘ * should. be asked and ‘that information ‘is regsived. 

: " ghtatning~doternining what data gathering’ strategy to use and 


by using this strategy to odllect data. “The strat- 


aw 


) ae a 
egy will sometimes involve the collection of "new" 
~, data while at other times it simply requires com- 


tie? pilation of data io existing sources. 


-providing--by feeding data back: ‘to the decision maker(s) and 


helping the decisfon maker(s) utilize the data to 

nese ‘the necessary TURESARIESs ; 

The Phi Delta Kapp Committee on Bvaluation (Stufflebeam et al., 

- 1971) has summed up the relationship besween decision maker and evalua- . 


tor under this definition as follows: . yt 


i S Although there is a close relationship between evaluation " 
and decision making, appropriate lines of demarcation must be z 
’ drawn between the two roles. Under special circumstances, the 3 
decision maker might perform many of his own: evaluation tasks ; 
. of delineating, obtaining, and using evaluative information. - 
, This-is good'to the extent that he can meet the criteria for 
adequate évaluation.. However, the. evaluator should by no fheans- 
assume responsibility for roles inherently those of the’ de- 
‘cision maker. The evaluator should neither make nor implement 
program-decisions. “The essence of his role is to provide infor-. 
mation: to' enable the decision maker to perform these tasks (p. 93). 


\ 4 . ; 5; 
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Summary 2 see ; \ 

I believe that educational evataspton as practiced in the ma~ ; 

. ority of instances bas failed *é provide practitioners with the in- a 
formation they have needed to salad education. . : 

_Much of the bl ales this fatlure of Syelnation to provide prac- - 
titdoners with necegfry information ean be laid sgudrely on the fail~ 
ure af evaluators té actively ingoive decision makers at every. sten of 
the evaluation process. ‘As, a result. of these failures, prackitioners,: ’ 
espsciaity those involved in developing aitarmative « educational Pro- * 
ais naw developed a skeptical (maybe even hostile) view of eval- . 


. vation. : ; 
2 ee ’ : . 


SEA educators believe that newly emerging alternative education&l eae 


j ° 


programs can benefit grea from formative evaluation information. 


oi aw as 


+. 


However, the evaluators providing the services must be credible in the 
eyes of the practitioners involved if the efforts are to succeed. To 
overcome past nebaehvinte toward evaluation and to develop credibility, 
evaluators must be aware ‘of and avoid past evaluation shortcomings. 
The present paper has outlined many of the more prevalent past éval- 
suite. akobooubiips and has discussed ways in which the SEA evaluation 
team believes they may reasonably be avoided. 
Evaluation should be considered as an integral part of the deci- 
sion making process. The specific decision to be made gives direc- 
tion-to the evaluation effort and evaluation provides information to 


aaatet in making the decision but it does not neKe the decision. The ~ 


\ evaluator helps the decision maker ad ineate the Untoradtton to be ee. 
' 


jnered, obtains the information, qnd, finally, provides the information te: 
. the decision makers. ro . : . 


' 
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